Selecting a Selection Procedure

Selection procedures are used in a variety of applications to select the best of a finite set of alternatives. ‘Best’ is defined with respect to the largest mean, but the mean is inferred with statistical sampling, as in simulation optimization. There are a wide variety of procedures, which begs the question of which selection procedure to select. The main contribution of this paper is to identify, through extensive experimentation, the most effective selection procedures when samples are independent and normally distributed. We also (a) summarize the main structural approaches to deriving selection procedures, (b) formalize new sampling allocations and stopping rules, (c) identify strengths and weaknesses of the procedures, (d) identify some theoretical links between them, (e) and present an innovative empirical test bed with the most extensive numerical comparison of selection procedures to date. The most efficient and easiest to control procedures allocate samples with a Bayesian model for uncertainty about the means, and use new adaptive stopping rules proposed here. Selection procedures are intended to select the best of a finite set of alternatives, where best is determined with respect to the largest mean, but the mean must be inferred via statistical sampling (Bechhofer et al. 1995). Selection procedures can inform managers how to select the best of a small set of alternative actions whose effects are evaluated with simulation (Nelson and Goldsman 2001), and have been implemented in commercial simulation products. Selection procedures have also attracted interest in combination with tools like multiple attribute utility theory (Butler et al. 2001), evolutionary algorithms (Branke and Schmidt 2004), and discrete optimization via simulation (Boesel et al. 2003). Three main approaches to solving the selection problem are distinguished by their assumptions about how the evidence for correct selection is described and sampling allocations are made: the indifference zone (IZ, Kim and Nelson 2006), the expected value of information procedure (VIP, Chick and Inoue 2001a), and the optimal computing budget allocation (OCBA, Chen 1996) approaches. IZ procedures typically allocate samples in order to provide a guaranteed lower bound for the frequentist probability of correct selection (PCS), with respect to the sampling distribution, for selection problems in a specific class (e.g., the mean of the best is at least a prespecified amount better than each alternative). The VIP approach describes the evidence for correct selection with Bayesian posterior distributions, and allocates further samples using decision-theory tools to maximize the expected value of information in those samples. The OCBA is a heuristic that uses a normal distribution approximation for the Bayesian posterior distribution of the unknown mean performance of each alternative in order to sequentially allocate further samples. Each approach stipulates a number of different sampling assumptions, approximations, stopping rules and parameters that combine to define a procedure. With so many variations, the question of which selection procedure to select arises. The question is important because the demands that are being placed upon simulation optimization algorithms are increasing. The answer may also provide new insights about differences in the empirical performance of three distinct approaches to statistical decision making (classical, or frequentist, statistics; Bayesian decision theory; and heuristic models about the probability of correct selection). ∗Institute AIFB, University of Karlsruhe, Germany {branke,csc}@aifb.uni-karlsruhe.de †INSEAD, Technology and Operations Management Area, [email protected] Sep 2005; Revised 4 May 2006; 11 Oct 2006; for Management Science 18 Jan 2007 p. 1 Branke, Chick, Schmidt Selecting a Selection Procedure A thorough comparison of these three approaches has not previously been done. Initial work shows that special cases of the VIP outperform specific IZ and OCBA procedures (in a comparison of two-stage procedures), and specific sequential VIP and OCBA procedures are more efficient than two-stage procedures (Inoue et al. 1999). TheKN family of procedures is effective among IZ procedures (Kim and Nelson 2006). No paper has studied more than a limited set of procedures with respect to a moderate experimental test bed. This paper addresses the unmet need for an extensive comparison of IZ, VIP and OCBA procedures. §1 summarizes the main approaches to selection procedures, derives new variants and formalizes new stopping rules for the VIP and OCBA procedures. Each procedure makes approximations, and none provides an optimal solution, so it is important to understand the strengths and weaknesses of each approach. §2 describes new measurements to evaluate each with respect to: • Efficiency: The mean evidence for correct selection as a function of the mean number of samples. • Controllability: The ease of setting a procedure’s parameters to achieve a targeted evidence level. • Robustness: The dependency of a procedure’s effectiveness on the underlying problem characteristics. • Sensitivity: The effect of the parameters on the mean number of samples needed. Some practitioners desire a (statistically conservative) lower bound for the targeted evidence level, such as a frequentist PCSIZ guarantee, but this may lead to excessive sampling. Together, efficiency and controllability indicate how close to the desired evidence level a procedure gets while avoiding excess sampling. The procedures are compared empirically on a large variety of selection problems described in §3. The test bed is unique not only because of its size, but also by its inclusion of randomized problem instances, in addition to structured problem instances that are usually studied, but that are unlikely to be found in practice. The focus is on applications where the samples are jointly independent and normally distributed with unknown and potentially different variances, or nearly so as is the case in stochastic simulation with batching (Law and Kelton 2000). Branke et al. (2005) presented a subset of preliminary empirical results, and assessed additional stopping rules that were somewhat less efficient than those considered below. §4 empirically compares the different selection procedures on a variety of test problems. The results show that a leading IZ procedure, called KN++ (described below), is more efficient than the original VIP and OCBA procedures, but is statistically conservative which may result in excessive sampling. In combination with the new stopping rules, the VIP and OCBA procedures are most efficient. They also tend to be more controllable and robust in the experiments below. §5 recommends those procedures, and discusses key issues for selecting a selection procedure. Appendices in the Online Companion generalize an OCBA procedure, give structural results that suggest why certain VIP and OCBA procedures perform similarly, describe the implementation, and display and interpret additional numerical results. 1 The Procedures We first formalize the problem, summarize assumptions and establish notation. §1.1 describes measures of the evidence of correct selection and, based thereon, introduces new stopping rules that improve efficiency. §1.2-1.4 describe existing and new procedures from the IZ, VIP and OCBA approaches. Sep 2005; Revised 4 May 2006; 11 Oct 2006; for Management Science 18 Jan 2007 p. 2 Branke, Chick, Schmidt Selecting a Selection Procedure The best of k simulated systems is to be identified, where ‘best’ means the largest output mean. Analogous results hold if smallest is best. Let Xij be a random variable whose realization xij is the output of the jth simulation replication of system i, for i = 1, . . . , k and j = 1, 2, . . .. Let wi and σ2 i be the unknown mean and variance of simulated system i, and let w[1] ≤ w[2] ≤ . . . ≤ w[k] be the ordered means. In practice, the ordering [·] is unknown, and the best system, system [k], is to be identified with simulation. The procedures considered below are derived from the assumption that simulation output is independent and normally distributed, conditional on wi and σ2 i , for i = 1, . . . , k. {Xij : j = 1, 2, . . .} iid ∼ Normal ( wi, σ 2 i ) Although the normality assumption is not always valid, it is often possible to batch a number of outputs so that normality is approximately satisfied. Vectors are written in boldface, such as w = (w1, . . . , wk) and σ2 = (σ2 1, . . . , σ 2 k). A problem instance (configuration) is denoted by χ = (w, σ 2). Let ni be the number of replications for system i run so far. Let x̄i = ∑ni j=1 xij/ni be the sample mean and σ̂2 i = ∑ni j=1(xij − x̄i)/(ni − 1) be the sample variance. Let x̄(1) ≤ x̄(2) ≤ . . . ≤ x̄(k) be the ordering of the sample means based on all replications seen so far. Equality occurs with probability 0 in contexts of interest here. The quantities ni, x̄i, σ̂2 i and (i) are updated as more replications are observed. Each selection procedure generates estimates ŵi of wi, for i = 1, . . . , k. For the procedures studied here, ŵi = x̄i, and a correct selection occurs when the selected system, system D, is the best system, [k]. Usually D = (k) is selected as best. If Tν is a random variable with standard t distribution with ν degrees of freedom, we denote (as do Bernardo and Smith 1994) the distribution of μ+ 1 √ κ Tν by St (μ, κ, ν). If ν > 2 the variance is κ−1ν/(ν−2). If κ = ∞ or 1/0, then St (μ, κ, ν) denotes a point mass at μ. Denote the cumulative distribution function (cdf) of the standard t distribution (μ = 0, κ = 1) by Φν() and probability density function (pdf) by φν(). 1.1 Evidence for Correct Selection This section provides a unified framework for describing both frequentist and Bayesian measures of selection procedure effectiveness and the evidence of correct selection. They are required to derive and compare the procedures below. They are also used within the Bayesian procedures (VIP and OCBA) to decide when the evidence of correct selection is sufficient to stop sampling. The measures are defined in terms of loss functions. The zero-one loss function, L0−1(D,w) = 1 { wD 6= w[k] } , equals 1 if the best system is not correctly selected, and is 0 otherwise. The opportunity cost Loc(D,w) = w[k]−wD is 0 if the best system is correctly selected, and is otherwise the difference between the best and selected system. The opportunity cost makes more sense in business applications. The IZ procedures take a frequentist perspective. The frequentist probability of correct selection (PCSIZ) is the probability that the mean of the system selected as best, system D equals the mean of the system with the highest mean, system [k], conditional on the problem instance (this allows for ties). The probability is with respect to the simulation output Xij generated by the procedure (the realizations xij determine D). PCSIZ(χ) def = 1− E [L0−1(D,w) |χ] = Pr ( wD = w[k] |χ ) Sep 2005; Revised 4 May 2006; 11 Oct 2006; for Management Science 18 Jan 2007 p. 3 Branke, Chick, Schmidt Selecting a Selection Procedure Indifference zone procedures attempt to guarantee a lower bound on PCSIZ, subject to the indifference-zone constraint that the best system is at least δ∗ > 0 better than the others, PCSIZ(χ) ≥ 1− α∗, for all χ = (w, σ2) such that w[k] ≥ w[k−1] + δ∗. (1) A selected system within δ∗ of the best is called good. Some IZ procedures satisfy frequentist probability of good selection guarantees, PGSIZ,δ∗(χ) def = Pr ( wD > w[k] − δ∗ |χ ) ≥ 1− α∗, for all configurations (Nelson and Banerjee 2001). Let PICSIZ = 1− PCSIZ and PBSIZ,δ∗ = 1− PGSIZ,δ∗ denote the probability of incorrect and bad selections. An alternative to a PCS guarantee for the evidence of correct selection is a guaranteed upper bound on the expected opportunity cost (EOC) of a potentially incorrect selection. The frequentist EOC (Chick and Wu 2005) is also defined with respect to the sampling distribution, EOCIZ(χ) def = E [Loc(D,w) |χ] = E [ w[k] − wD |χ ] . Bayesian procedures assume that parameters whose values are unknown are random variables (such as the unknown means W), and use the posterior distributions of the unknown parameters to measure the quality of a selection. Given the data E seen so far, two measures of selection quality are PCSBayes def = 1− E [L0−1(D,W) | E ] = Pr ( WD ≥ max i6=D Wi | E ) EOCBayes def = E [Loc(D,W) | E ] = E [ max i=1,2,...,k Wi −WD | E ] , (2) the expectation taken over both D (which is determined by the random Xij) and the posterior distribution of W, given E . Assuming a noninformative prior distribution for the unknown mean and variance, the posterior marginal distribution for the unknown mean Wi, given ni > 2 samples, is St ( x̄i, ni/σ̂ 2 i , νi ) , where νi = ni− 1 (de Groot 1970). Each Bayesian procedure below selects the system with the best sample mean after all sampling is done, D = (k). Approximations in the form of bounds on the above losses are useful to derive sampling allocations and to define stopping rules. Slepian’s inequality (e.g., see Kim and Nelson 2006) implies that the posterior evidence that system (k) is best satisfies PCSBayes ≥ ∏ j:(j)6=(k) Pr ( W(k) > W(j) | E ) . (3) The right hand side of Inequality (3) is approximately PCSSlep = ∏ j:(j)6=(k) Φν(j)(k)(d ∗ jk), where djk is the normalized distance for systems (j) and (k), and ν(j)(k) comes from Welch’s approximation for the difference W(k) −W(j) of two shifted and scaled t random variables (Law and Kelton 2000, p. 559): djk = d(j)(k)λ 1/2 jk with d(j)(k) = x̄(k) − x̄(j) and λ−1 jk = σ̂2 (j) n(j) + σ̂2 (k) n(k) , (4) ν(j)(k) = [σ̂2 (j)/n(j) + σ̂ 2 (k)/n(k)] 2 [σ̂2 (j)/n(j)] /(n(j) − 1) + [σ̂2 (k)/n(k)]/(n(k) − 1) . Sep 2005; Revised 4 May 2006; 11 Oct 2006; for Management Science 18 Jan 2007 p. 4 Branke, Chick, Schmidt Selecting a Selection Procedure We found that the Welch approximation outperformed another approximation in earlier comparisons of selection procedures (Branke et al. 2005). The Bayesian posterior probability of a good selection, where the selected system is within δ∗ of the best, can be approximated in a similar manner by